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(54) Situation awareness system 

(57) A situation awareness system includes a plu- 
rality of cameras. Each camera acquires a sequence of 
images of a particular part of an area of interest. There 
is overlap between the parts so that the system can 
obtain depth information about objects in the area of 
interest. An analyzer identifies moving objects in the 



areas of interest, attributes of the moving objects, and 
events related to the moving objects. A display device 
displays the attributed objects and events as annotated 
graphic elements and alerts. 
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Description 

FIELD OF THE INVENTION 

[0001] The invention relates generally to a monitor- 5 
ing system rendering a synthetic display derived from 
multiple cameras mounted at various locations, and 
more particularly, to alerting a viewer on special situa- 
tions observed, 

BACKGROUND OF THE INVENTION 

[0002] Video monitoring systems are well known. In 
the case of vehicles, several types of monitoring sys- 
tems are in use. Some vehicles, e.g., busses, have 
cameras mounted so that the driver can view road 
areas beside or behind the bus. However, there is typi- 
cally only one camera, and the display merely shows 
exactly what the camera sees. There is no attempt to 
analyze the displayed image. These systems simply act 
as viewing mirrors lor hard-to-see areas. Similarly, law 
enforcement vehicles may capture a historical record of 
the view from the front window. 
[0003] Some vehicles, such as computer controlled 
cars, also include sensors. The sensors detect poten- 
tially dangerous situations, such as, closing-up too rap- 
idly on another vehicle. A variety of sensors have been 
used, for example, sonar, lasers, and microwaves. 
These systems do not provide a general situation dis- 
play, rather they merely detect a few dangerous situa- 
tions, 

[0004] Radar and sonar systems can produce a sit- 
uation display, and sometimes do some amount of anal- 
ysis, for example, as in an air traffic control system. 
However, radar and sonar systems are not based on 
video images, but rather on the processing of reflected 
signals transmitted at specific frequencies. 
[0005] Several types of surveillance systems are 
known. Typically, the systems route multiple video 
streams to a central location. The video streams can be 
displayed on corresponding monitors. If the number of 
cameras is greater than the number of display stations, 
then the system usually displays camera views in 
sequence, or on operator demand. These type of sys- 
tems do not perform analysis, nor do these systems 
merge multiple streams into a single situation display. At 
most they may tile multiple independent views on a sin- 
gle screen with time and location annotations. 
[0006] There are also systems that monitor specific 
places, such as escalators, elevators, toll gates, bank 
machines, and perimeter fences, in order to determine 
the occurrence of particular situations. Some of these 
systems may attempt to analyze the video in order to 
detect moving objects, for example, to extract a license 
number. However, these system typically do not com- 
bine information from multiple sources, nor do they gen- 
erate an overall situation display, nor synthesize an 
image from a different point of view. 
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SUMMARY OF THE INVENTION 

[0007] The invention provides a situation aware- 
ness system which includes a plurality of cameras. 
Each camera acquires a sequence of images of over- 
lapping parts of an area of interest An analyzer merges 
the sequences of images acquired by the plurality of 
cameras, and identifies moving objects in the area of 
interest. A display device displays the merged 
sequences of images, and information associated with 
the identified moving objects. 

[0008] In one aspect of the invention, the optical 
flow in temporally successive images of a single video 
stream are analyzed to generate motion fields. Spatially 
adjacent images of multiple video stream are registered 
to obtain depth images. The motion fields and depth 
images are segmented to generate partially attributed 
data objects. Using an application specific database 
and analysis, the partially attributed data objects are 
converted to fully attributed data objects and events 
which are displayed as annotated graphic elements and 
alerts. As one feature of the invention, the viewing orien- 
tation of the display is independent of the point of view 
of the cameras. 

BRIEF DESCRIPTION OF THE DRAWINGSFigure 1 is 

a block diagram of an awareness system according to 
the invention; 

[0009] 

Figure 2 is a block diagram of an analyzer synthe- 
sizer of the system of Figure 1 ; and 
Figure 3 is an example synthetic image generated 
by the system of Figure 1 . 

DETAILED DESCRIPTION OF PREFERRED EMBOD- 
IMENTSSystem Overview 

[001 0] Figure 1 shows the situation awareness sys- 
tem 100 according to my invention. The system 100 
includes multiple video cameras 101-106. Each camera 
acquires a sequence of images as a video stream 115. 
Six cameras are shown, fewer or more can be used. 
Additional cameras can be provided for redundancy in 
the case of a camera failure, or obstruction. The cam- 
eras can be arranged to obtain a full 360 degree field of 
view of an area of interest around a vehicle. 
[001 1 J For other applications, a smaller field of view 
is suitable. The images provided by each camera over- 
lap parts of the area of interest such that stereoscopic 
techniques can be used to extract depth information. 
Wide angle lenses can be used to increase the amount 
of overlap without increasing the number of cameras. 
[0012] The output of the cameras, digitized video 
streams 115, is connected to an analyzer-synthesizer 
200. The analyzer-synthesizer 200, according to my 
invention, analyzes the video streams and generates a 
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synthetic display 300 on an output device 1 20. 
System Operation 

[0013] In an operational system, the cameras can 
be mounted on, for example, a vehicle 130 shown by 
dashed lines in Figure 1. The cameras can also be 
placed at other fixed or moving locations to observe the 
area of interest, generally 125, the areas in front of the 
various lenses. 

[0014] The analyzer-synthesizer 200 operates on 
the data of the multiple video streams in real -time. The 
analyzer portion extracts temporal and spatial data from 
the video streams to identify objects, and their 
attributes, such as size, position, and velocity In addi- 
tion, relationships between the identified objects are 
determined, for example, two vehicles on intersecting 
courses. In other words the video streams are reduced 
to a relationship of attributed objects. The attributed 
objects are analyzed to detect events, for example, a 
possible collision, or a danger zone. The synthesizer 
portion generates the situation awareness display 300 
of the relationships of the attributed objects, and 
optional alerts related to the events. 
[0015] According to my invention, the situation 
awareness display 300 is entirely synthetic. In contrast 
with the prior art, I discard the video stream 1 15 after it 
is analyzed. In addition, the synthesizer integrates infor- 
mation extracted from the multiple video streams into a 
single display 300, Furthermore, aiert signals 140 may 
be generated when certain dangerous situations or 
events are recognized. The alert signals can be dis- 
played, or presented to some other output device 150, 
In an alternative embodiment, the alert signals 140 can 
initiate evasive collision avoidance action, for example, 
braking or slowing down. 

Synthesizer-Analyzer 

10016] As shown in Figure 2, video streams 115 
from multiple cameras 101-106 are presented to the 
analyzer/synthesizer 200, via an A/D converter if neces- 
sary, as digital video data 201. Temporal and spatial 
information is extracted from the digital video data 201 . 
[0017] Optical flow analysis 210 is used to deter- 
mine motion fields 21 1 from images separated in time 
(?t), for example, from motion fields of successive 
frames in a single video sequence. 
[0018] Image registration 220 is used to determine 
a depth image 221 from images overlapping in space 
(?x, ?y), for example, using frames taken of overlapping 
parts of the area of interest by multiple cameras. The 
depth image specifies the distance (?z) to each pixel in 
the image. 

[001 9] The motions fields and depth image are seg- 
mented to produce partially attributed data objects 231 . 
For example, pixels having the same optica) flow at the 
same depth are likely to be related to the same object. 



Usrng both the optical flow and distances provides for a 
robust segmentation, particularly when the flow analy- 
sis is done concurrently with the registration so the 
derived results (motion fields and depth values) corre- 

5 late with each other (215). 

[0020] The partial attributes can include the size, 
position, velocity, direction of movement of the objects in 
thee-dimensional space. The objects are only partially 
attributed because other attributes that depend on addi- 

10 tional knowledge, such as the exact identity of the 
objects have not yet been determined. 
[0021] The partially attributed data objects 231, in 
conjunction with an application specific database 239 
can be analyzed 240 to generate fully attributed data 

is objects 241 and events 242. For example, a one-sided 
view of a semi-trailer is sufficient to deduce the entire 
shape of the object. Various kinds of template matching 
schemes can be used to fully identify specific commonly 
occurring objects, such as F other vehicles, pedestrians, 

20 bicycles, trucks, and the like. In a vehicle application, 
the features may also include lane dividers, side walks, 
stop signs, guard rails, curbs, buildings, fences, and so 
forth. 

[0022] The events 242 can be generated by analyz- 
es ing the relationships among the attributed objects, for 
example, a potential collision situation, a car drifting off 
the road, or a fading light situation. Additional sensors 
249 can also be used to enrich the number of events 
that can be detected. 
30 [0023] A synthesizer 250 converts the fully attrib- 
uted data objects 241 to annotated graphic elements 
251 and alerts 252. The last step renders 260 the 
graphic elements 251 and alerts 252. 



[0024] Many different types of situation displays are 
possible. The display 300 in Figure 3 shows a bird's eye 
view of the area of interest with the vehicle 31 0 on which 
the situation awareness device is mounted, located at a 
fixed orientation near the center of the display, and 
annotated objects moving relative to the point of view. 
Note, the view is totally synthetic and orthogonal to the 
view seen by the cameras. 

[0025] Certain other image features are shown as 
welt, such as pedestrian lane crossing 320, buildings 
330, other traffic 340, a bicycle 350, and so forth. 
[0026] Arrows 301 can be used to show the direc- 
tion of movement of objects that are not stationary. 
Determining the orientation of the arrows requires an 
active analysis, as opposed to passively displaying the 
output of the cameras as done in the prior art. 
In an area of interest where sufficient ambient light can 
not be assured, my invention can be extended by includ- 
ing active illumination. In some situations it could benefit 
from using infrared light, either to see in the dark without 
requiring active illumination or as inoffensive active illu- 
mination. In situations such as fog, where visibility is 
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poor, my invention can operate by carefully selected 
wavelengths or strobed light sources appropriately syn- 
chronized with the shutter of the cameras so as to focus 
on objects of interest and reject other scattered light. 
[0027] In one embodiment of my invention, the ana- 5 
lyzing step 240 can receive secondary data 238. In a 
vehicle application, the data Gan include vehicle velocity, 
or position as obtained from a GPS receiver. With the 
vehicle's velocity, the analysis can be improved and sim- 
plified. Positional data enables the use of maps on the 10 
display, and actual street and place names. 
[0028] In another embodiment, the display 300 is 
under user control. For instance, in a building surveil- 
lance application, the user supply control signals 239 to 
alter the way that the annotated graphic elements and is 
alerts are displayed, or to change the orientation of the 
point of view. It is also possible to transmit the alerts and 
graphic elements to a remote location. For instance, 
while walking toward a parked vehicle, the operator can 
view, on a portable display device, the area of interest in 20 
the vicinity of the vehicle from a safe, location. 
[0029] In addition, multiple vehicles can exchange 
situation information with each other to enhance the 
scope of the display. Other areas where the invention 
can be used include airports, waterways, and the like, 25 
[0030] This Invention is described using specific 
terms and examples. It is to be understood that various 
other adaptations and modifications may be made 
within the spirit and scope of the invention. Therefore, it 
is the object of the appended claims to cover all such 30 
variations and modifications as come within the true 
spirit and scope of the invention. 

Claims 

35 

1. A real-time situation awareness system, compris- 
ing: 

a plurality of cameras acquiring a plurality of 
video streams of overlapping parts of an area *o 
of interest; 

analyzer means for reducing the plurality of 
video streams to attributed data objects and 
events; and 

synthesizer means for rendering the attributed 45 
data objects and events as annotated graphic 
elements and alerts on an output device, 

2. The system of claim 1 further comprising: 

50 

means for temporally analyzing an optical flow 
in successive images of a single video stream 
to generate motion fields; 
means for spatially registering adjacent images 
of multiple video stream to obtain depth ss 
images; and 

means for segmenting the motion fields and 
depth images to generate partially attributed 
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data objects. 

3. The system of claim 2 further comprising: 

means for analyzing the partially attributed 
data objects using an application specif ic data- 
base to generate fully attributed data objects 
and events. 

4. The system of claim 3 further comprising: 

sensors providing the analyzing step with sec- 
ondary data and signals. 

5. The system of claim 1 wherein the synthesizer 
means produces a display having a point of view 
substantially orthogonal to the point of view of the 
cameras. 

6. The system of claim 1 wherein the area of interest 
is a panoramic scene. 

7. The system of claim 1 wherein annotations for the 
graphic elements include directions of movement 

8. The system of claim 5 wherein user control signals 
determine the display. 

9. A method for generating a reaMime situation 
awareness display comprising the steps of: 

acquiring a plurality of video streams of over- 
lapping parts of an area of interest; 
reducing the plurality of video streams to attrib- 
uted data objects and events; and 
rendering the attributed data objects and 
events as annotated graphic elements and 
alerts on an output device. 

1 0. The method of claim 9 further comprising: 

temporally analyzing an optical flow in succes- 
sive images of a single video stream to gener- 
ate motion fields; 

spatially registering adjacent images of multi- 
ple video stream to obtain depth images; and 
segmenting the motion f ields and depth images 
to generate partially attributed data objects, 

1 1 . The method of claim 1 0 further comprising: 

analyzing the partially attributed data objects 
using an application specific database to gen- 
erate fully attributed data objects and event. 



EP1 030188 A1 



4 



ISDOCID:<EP ..10301S8A1_I_> 



EP1 030 188 A1 




5 

JSDOCID:<EP 103018SA1 I > 



EP1 030 188 A1 




6 

SDOCID: <EP 1 03018BA1 



1 EP1 030 188 A1 




^SDOCID <EP 10301B8A1 I > 



7 



EP1 030 188 A1 



European Patent 
Offlc* 



EUROPEAN SEARCH REPORT 



Application Number 

EP 00 10 2979 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate, 



Relevant 
to ctakn 



CLASSIFICATION OF THE 
APPLICATION OnLCLT) 



US 5 410 346 A (SANEYOSHI KEIJI ET AL) 
25 April 1995 (1995-04-25) 

* column 7, line 13 - column 8, line 22; 
figures 1,3,12,32-39 * 

* column 28, line 5 - column 32, line 6 * 

W0 98 45816 A (SYNAPIX INC) 
15 October 1998 (1998-10-15) 

* page 11, line 26 - page 13, line 22; 
figures 1,2,7,8 * 

* page 29, line 22 - page 41, line 17 * 

US 5 729 216 A (FUJINAMI KAZUT0MO ET AL) 
17 March 1998 (1998-03-17) 

* column 3, line 22 - column 4, line 15; 
figures 1,2 * 

EP 0 626 654 A (MATSUSHITA ELECTRIC IND CO 
LTD) 30 November 1994 (1994-11-30) 

* page 3, line 9 - page 3, line 32; figure 
1 * 

US 5 530 420 A (HANAUA KEIJI ET AL) 
25 June 1996 (1996-06-25) 

* column 4, line 1 - column 5, line 18; 
figures 1,2,13,35-39 * 

* column 21, line 46 - column 23, line 47 
* 



The present search report ha8 been drawn up tor all claims 



1-11 



1-11 



1,9 



1,9 



1,9 



601S11/12 
606T7/00 



TECHMCAL RBJDS 
SEARCHED (M.CL7) 



eois 

G06T 



MUNICH 



CMb at ccrrptotton of th* MKh 

25 May 2000 



Cannard, J-M 



CATEGORY OF CfTED DOCUMENTS 

X r parttoiarty ralavant Kiakan ak»» 
Y : parttaiarty ralavant if combined wth anothor 
dbourwnt of tha aania category 

A i t i n h n ll I n i Ji i ■! j i ■ i ll i ■ ii 11 nw4 

n. - (wnnQtQiPCcM UfK*^ UUfKJ 

O: 



T t thaofy or princtpto undarlylrtQ tha mvantfon 
E i Mffar patent dpcumant. but putolanad on, of 

sftof tha flllnQ data 
C r document cited In the application 
L : docunant dted for otfwr reeeona 

& ; rnaniiw of tha earn* patent Canity, oorreapondng 



8 



3DOCID: <EP. 



.10301BBA1 I > 



EP1 030 188 A1 



ANNEX TO THE EUROPEAN SEARCH REPORT 

ON EUROPEAN PATENT APPLICATION NO. EP 00 10 2979 



This annex Seta (he patent family mam bens rsiadng to tfia patent documents cited in the above-mentioned Europem search report. 
The rnemfrers are aa contained in the European Patent Office EDP me on 

The European Patent Office is in no way labia for these parfculare which are merely given for the purpose of information, 

25-05-2000 



Patent document 


Publication 


Patent famly 

mamberta) 


Publication 


cited In search report 


date 




US 5410346 A 


25-04-1995 


JP 


5265547 A 


15-10-1993 






DE 


4308776 A 


30-09-1993 






GB 


2265779 A,B 


06-10-1993 1 



W0 9845816 



15-10-1998 



AU 


6791298 


A 


30-10-1998 


All 


6874898 


A 


30-10-1998 


AU 


6876298 


A 


30-10-1998 


AU 


6944098 


A 


30-10-1998 


EP 


0974122 


A 


26-01-2000 


EP 


0974123 


A 


26-01-2000 


EP 


0974124 


A 


26-01-2000 


EP 


0974126 


A 


26-01-2000 


WO 


9845812 


A 


15-10-1998 


WO 


9845813 


A 


15-10-1998 


WO 


9845814 


A 


15-10-1998 



US 


5729216 


A 


17-03- 


-1998 


OP 


7250319 A 


26-09-1995 


EP 


0626654 


A 


30-11- 


-1994 


JP 


6331351 A 


02-12-1994 


US 


5530420 


A 


25-06- 


-1996 


JP 


7192199 A 


28-07-1995 












DE 


4446452 A 


29-06-1995 












GB 


2285719 A,B 


19-07-1995 



I 

I 

J? 

Si For mors details about Ihis annex : sm Official Journal of the European Patent Office, No. 12/82 



9 



ISDOCID: <EP. 1 0301 6BA1 _l_> 



